{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第六周作业"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "问题描述\n",
    "使用tensorflow，构造并训练一个神经网络，在测试机上达到超过98%的准确率。\n",
    "\n",
    "解题提示\n",
    "在完成过程中，需要综合运用目前学到的基础知识：\n",
    "1.深度神经网络\n",
    "2.激活函数\n",
    "3.正则化\n",
    "4.初始化\n",
    "\n",
    "并探索如下超参数设置：\n",
    "1.隐层数量\n",
    "2.各隐层中神经元数量\n",
    "3.学习率\n",
    "4.正则化因子\n",
    "5.权重初始化分布参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\h5py\\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    }
   ],
   "source": [
    "\"\"\"A very simple MNIST classifier.\n",
    "See extensive documentation at\n",
    "https://www.tensorflow.org/get_started/mnist/beginners\n",
    "\"\"\"\n",
    "from __future__ import absolute_import\n",
    "from __future__ import division\n",
    "from __future__ import print_function\n",
    "\n",
    "import argparse\n",
    "import sys\n",
    "\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "FLAGS = None\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们在这里调用系统提供的Mnist数据函数为我们读入数据，如果没有下载的话则进行下载。\n",
    "\n",
    "<font color=#ff0000>**这里将data_dir改为适合你的运行环境的目录**</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting /tmp/tensorflow/mnist/input_data\\train-images-idx3-ubyte.gz\n",
      "Extracting /tmp/tensorflow/mnist/input_data\\train-labels-idx1-ubyte.gz\n",
      "Extracting /tmp/tensorflow/mnist/input_data\\t10k-images-idx3-ubyte.gz\n",
      "Extracting /tmp/tensorflow/mnist/input_data\\t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "# Import data\n",
    "data_dir = '/tmp/tensorflow/mnist/input_data'\n",
    "mnist = input_data.read_data_sets(data_dir, one_hot=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the model\n",
    "#batch_size = 50\n",
    "#n_batch = mnist.train.num_examples\n",
    "x = tf.placeholder(tf.float32, [None, 784])\n",
    "y = tf.placeholder(tf.float32, [None, 10])\n",
    "#print(n_batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#定义神经网络中间层\n",
    "W_L1 = tf.Variable(tf.random_normal([784,100]))\n",
    "b_L1 = tf.Variable(tf.constant(0.001,shape=[100])) \n",
    "Wx_L1 = tf.matmul(x, W_L1)+b_L1 \n",
    "L1 = tf.nn.sigmoid(Wx_L1) #使用正切函数作为激活函数 \n",
    "\n",
    "W_L2 = tf.Variable(tf.random_normal([100,100]))\n",
    "b_L2 = tf.Variable(tf.constant(0.001,shape=[100])) \n",
    "Wx_L2 = tf.matmul(L1, W_L2)+b_L2 \n",
    "L2 = tf.nn.elu(Wx_L2)#用正切函数作为激活函数 \n",
    "\n",
    "#W_L3= tf.Variable(tf.random_normal([100,100]))\n",
    "#b_L3= tf.Variable(tf.constant(0.001,shape=[100])) \n",
    "#Wx_L3= tf.matmul(L2,W_L3)+b_L3\n",
    "#L3= tf.nn.elu(Wx_L3)#使用正切函数作为激活函数 "
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "在这里尝试了增加隐层数量和增加神经元数量，并且也试过不同的赋值（zeros、ones和0.001等）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义我们的ground truth 占位符"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "W_L3 = tf.Variable(tf.random_normal([100,10]))\n",
    "b_L3 = tf.Variable(tf.constant(0.001,shape=[10]))\n",
    "Wx_L3 = tf.matmul(L2,W_L3) + b_L3\n",
    "prediction = tf.nn.tanh(Wx_L3)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "同时也尝试了不同类型的激活函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define loss and optimizer\n",
    "#y_ = tf.placeholder(tf.float32, [None, 10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来我们计算交叉熵，注意这里不要使用注释中的手动计算方式，而是使用系统函数。\n",
    "另一个注意点就是，softmax_cross_entropy_with_logits的logits参数是**未经激活的wx+b**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The raw formulation of cross-entropy,\n",
    "#\n",
    "#   tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(tf.nn.softmax(y)),\n",
    "#                                 reduction_indices=[1]))\n",
    "#\n",
    "# can be numerically unstable.\n",
    "#\n",
    "# So here we use tf.nn.softmax_cross_entropy_with_logits on the raw\n",
    "# outputs of 'y', and then average across the batch.\n",
    "#cross_entropy = tf.reduce_mean(\n",
    " #   tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "生成一个训练step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss = tf.reduce_mean(tf.square(y-prediction))\n",
    "#train_step = tf.train.AdamOptimizer(0.7).minimize(loss)\n",
    "train_step = tf.train.GradientDescentOptimizer(0.5).minimize(loss)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "学习率也进行了多次的调整"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在这里我们仍然调用系统提供的读取数据，为我们取得一个batch。\n",
    "然后我们运行3k个step(5 epochs)，对权重进行优化。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train\n",
    "init = tf.global_variables_initializer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "验证我们模型在测试数据上的准确率"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter0,Testing Accuracy0.7341\n",
      "Iter1,Testing Accuracy0.7196\n",
      "Iter2,Testing Accuracy0.7701\n",
      "Iter3,Testing Accuracy0.792\n",
      "Iter4,Testing Accuracy0.7991\n",
      "Iter5,Testing Accuracy0.8554\n",
      "Iter6,Testing Accuracy0.9038\n",
      "Iter7,Testing Accuracy0.8693\n",
      "Iter8,Testing Accuracy0.9086\n",
      "Iter9,Testing Accuracy0.9098\n",
      "Iter10,Testing Accuracy0.9188\n",
      "Iter11,Testing Accuracy0.9384\n",
      "Iter12,Testing Accuracy0.9426\n",
      "Iter13,Testing Accuracy0.95\n",
      "Iter14,Testing Accuracy0.9446\n",
      "Iter15,Testing Accuracy0.9524\n",
      "Iter16,Testing Accuracy0.9519\n",
      "Iter17,Testing Accuracy0.9508\n",
      "Iter18,Testing Accuracy0.953\n",
      "Iter19,Testing Accuracy0.9561\n",
      "Iter20,Testing Accuracy0.9559\n",
      "Iter21,Testing Accuracy0.9543\n",
      "Iter22,Testing Accuracy0.9559\n",
      "Iter23,Testing Accuracy0.9495\n",
      "Iter24,Testing Accuracy0.9574\n",
      "Iter25,Testing Accuracy0.9575\n",
      "Iter26,Testing Accuracy0.9558\n",
      "Iter27,Testing Accuracy0.9566\n",
      "Iter28,Testing Accuracy0.9549\n",
      "Iter29,Testing Accuracy0.9568\n",
      "Iter30,Testing Accuracy0.9535\n",
      "Iter31,Testing Accuracy0.955\n",
      "Iter32,Testing Accuracy0.9577\n",
      "Iter33,Testing Accuracy0.9502\n",
      "Iter34,Testing Accuracy0.9559\n",
      "Iter35,Testing Accuracy0.9535\n",
      "Iter36,Testing Accuracy0.9568\n",
      "Iter37,Testing Accuracy0.9568\n",
      "Iter38,Testing Accuracy0.955\n",
      "Iter39,Testing Accuracy0.9579\n",
      "Iter40,Testing Accuracy0.9542\n",
      "Iter41,Testing Accuracy0.9568\n",
      "Iter42,Testing Accuracy0.957\n",
      "Iter43,Testing Accuracy0.9557\n",
      "Iter44,Testing Accuracy0.9565\n",
      "Iter45,Testing Accuracy0.9546\n",
      "Iter46,Testing Accuracy0.9558\n",
      "Iter47,Testing Accuracy0.9513\n",
      "Iter48,Testing Accuracy0.9539\n",
      "Iter49,Testing Accuracy0.9543\n",
      "Iter50,Testing Accuracy0.9523\n",
      "Iter51,Testing Accuracy0.9495\n",
      "Iter52,Testing Accuracy0.957\n",
      "Iter53,Testing Accuracy0.9579\n",
      "Iter54,Testing Accuracy0.9579\n",
      "Iter55,Testing Accuracy0.9553\n",
      "Iter56,Testing Accuracy0.9515\n",
      "Iter57,Testing Accuracy0.9544\n",
      "Iter58,Testing Accuracy0.9574\n",
      "Iter59,Testing Accuracy0.9518\n",
      "Iter60,Testing Accuracy0.9525\n",
      "Iter61,Testing Accuracy0.9536\n",
      "Iter62,Testing Accuracy0.9557\n",
      "Iter63,Testing Accuracy0.9565\n",
      "Iter64,Testing Accuracy0.9573\n",
      "Iter65,Testing Accuracy0.9529\n",
      "Iter66,Testing Accuracy0.9587\n",
      "Iter67,Testing Accuracy0.9575\n",
      "Iter68,Testing Accuracy0.9576\n",
      "Iter69,Testing Accuracy0.9598\n",
      "Iter70,Testing Accuracy0.9589\n",
      "Iter71,Testing Accuracy0.9543\n",
      "Iter72,Testing Accuracy0.9568\n",
      "Iter73,Testing Accuracy0.9519\n",
      "Iter74,Testing Accuracy0.9563\n",
      "Iter75,Testing Accuracy0.9563\n",
      "Iter76,Testing Accuracy0.9552\n",
      "Iter77,Testing Accuracy0.9566\n",
      "Iter78,Testing Accuracy0.9534\n",
      "Iter79,Testing Accuracy0.9573\n",
      "Iter80,Testing Accuracy0.9549\n",
      "Iter81,Testing Accuracy0.9567\n",
      "Iter82,Testing Accuracy0.9575\n",
      "Iter83,Testing Accuracy0.9548\n",
      "Iter84,Testing Accuracy0.9566\n",
      "Iter85,Testing Accuracy0.9558\n",
      "Iter86,Testing Accuracy0.9551\n",
      "Iter87,Testing Accuracy0.9579\n",
      "Iter88,Testing Accuracy0.9577\n",
      "Iter89,Testing Accuracy0.9537\n",
      "Iter90,Testing Accuracy0.9531\n",
      "Iter91,Testing Accuracy0.9582\n",
      "Iter92,Testing Accuracy0.9545\n",
      "Iter93,Testing Accuracy0.9541\n",
      "Iter94,Testing Accuracy0.9549\n",
      "Iter95,Testing Accuracy0.9588\n",
      "Iter96,Testing Accuracy0.9578\n",
      "Iter97,Testing Accuracy0.9601\n",
      "Iter98,Testing Accuracy0.9593\n",
      "Iter99,Testing Accuracy0.9604\n",
      "Iter100,Testing Accuracy0.9589\n",
      "Iter101,Testing Accuracy0.9604\n",
      "Iter102,Testing Accuracy0.9586\n",
      "Iter103,Testing Accuracy0.9603\n",
      "Iter104,Testing Accuracy0.9589\n",
      "Iter105,Testing Accuracy0.9579\n",
      "Iter106,Testing Accuracy0.9592\n",
      "Iter107,Testing Accuracy0.9595\n",
      "Iter108,Testing Accuracy0.9569\n",
      "Iter109,Testing Accuracy0.9595\n",
      "Iter110,Testing Accuracy0.9579\n",
      "Iter111,Testing Accuracy0.9603\n",
      "Iter112,Testing Accuracy0.9588\n",
      "Iter113,Testing Accuracy0.9587\n",
      "Iter114,Testing Accuracy0.9603\n",
      "Iter115,Testing Accuracy0.9601\n",
      "Iter116,Testing Accuracy0.96\n",
      "Iter117,Testing Accuracy0.9584\n",
      "Iter118,Testing Accuracy0.9598\n",
      "Iter119,Testing Accuracy0.9578\n",
      "Iter120,Testing Accuracy0.9603\n",
      "Iter121,Testing Accuracy0.9596\n",
      "Iter122,Testing Accuracy0.9593\n",
      "Iter123,Testing Accuracy0.9572\n",
      "Iter124,Testing Accuracy0.9571\n",
      "Iter125,Testing Accuracy0.9585\n",
      "Iter126,Testing Accuracy0.9594\n",
      "Iter127,Testing Accuracy0.9597\n",
      "Iter128,Testing Accuracy0.9583\n",
      "Iter129,Testing Accuracy0.9605\n",
      "Iter130,Testing Accuracy0.9583\n",
      "Iter131,Testing Accuracy0.9589\n",
      "Iter132,Testing Accuracy0.9586\n",
      "Iter133,Testing Accuracy0.9614\n",
      "Iter134,Testing Accuracy0.9587\n",
      "Iter135,Testing Accuracy0.9604\n",
      "Iter136,Testing Accuracy0.9563\n",
      "Iter137,Testing Accuracy0.9581\n",
      "Iter138,Testing Accuracy0.9595\n",
      "Iter139,Testing Accuracy0.9598\n",
      "Iter140,Testing Accuracy0.9582\n",
      "Iter141,Testing Accuracy0.9598\n",
      "Iter142,Testing Accuracy0.9554\n",
      "Iter143,Testing Accuracy0.96\n",
      "Iter144,Testing Accuracy0.961\n",
      "Iter145,Testing Accuracy0.9589\n",
      "Iter146,Testing Accuracy0.9591\n",
      "Iter147,Testing Accuracy0.9596\n",
      "Iter148,Testing Accuracy0.9586\n",
      "Iter149,Testing Accuracy0.9601\n",
      "Iter150,Testing Accuracy0.9596\n",
      "Iter151,Testing Accuracy0.96\n",
      "Iter152,Testing Accuracy0.9604\n",
      "Iter153,Testing Accuracy0.9591\n",
      "Iter154,Testing Accuracy0.9562\n",
      "Iter155,Testing Accuracy0.9569\n",
      "Iter156,Testing Accuracy0.9589\n",
      "Iter157,Testing Accuracy0.9599\n",
      "Iter158,Testing Accuracy0.9572\n",
      "Iter159,Testing Accuracy0.9593\n",
      "Iter160,Testing Accuracy0.9604\n",
      "Iter161,Testing Accuracy0.9554\n",
      "Iter162,Testing Accuracy0.9577\n",
      "Iter163,Testing Accuracy0.9596\n",
      "Iter164,Testing Accuracy0.9591\n",
      "Iter165,Testing Accuracy0.9591\n",
      "Iter166,Testing Accuracy0.9603\n",
      "Iter167,Testing Accuracy0.9599\n",
      "Iter168,Testing Accuracy0.9598\n",
      "Iter169,Testing Accuracy0.9613\n",
      "Iter170,Testing Accuracy0.9601\n",
      "Iter171,Testing Accuracy0.9592\n",
      "Iter172,Testing Accuracy0.9603\n",
      "Iter173,Testing Accuracy0.961\n",
      "Iter174,Testing Accuracy0.9571\n",
      "Iter175,Testing Accuracy0.9601\n",
      "Iter176,Testing Accuracy0.9601\n",
      "Iter177,Testing Accuracy0.9579\n",
      "Iter178,Testing Accuracy0.9608\n",
      "Iter179,Testing Accuracy0.9601\n",
      "Iter180,Testing Accuracy0.9596\n",
      "Iter181,Testing Accuracy0.9606\n",
      "Iter182,Testing Accuracy0.9603\n",
      "Iter183,Testing Accuracy0.9596\n",
      "Iter184,Testing Accuracy0.96\n",
      "Iter185,Testing Accuracy0.954\n",
      "Iter186,Testing Accuracy0.9581\n",
      "Iter187,Testing Accuracy0.9581\n",
      "Iter188,Testing Accuracy0.9585\n",
      "Iter189,Testing Accuracy0.9581\n",
      "Iter190,Testing Accuracy0.9607\n",
      "Iter191,Testing Accuracy0.9586\n",
      "Iter192,Testing Accuracy0.9606\n",
      "Iter193,Testing Accuracy0.9574\n",
      "Iter194,Testing Accuracy0.9568\n",
      "Iter195,Testing Accuracy0.9587\n",
      "Iter196,Testing Accuracy0.96\n",
      "Iter197,Testing Accuracy0.9603\n",
      "Iter198,Testing Accuracy0.9607\n",
      "Iter199,Testing Accuracy0.9544\n",
      "Iter200,Testing Accuracy0.9584\n",
      "Iter201,Testing Accuracy0.9601\n",
      "Iter202,Testing Accuracy0.9594\n",
      "Iter203,Testing Accuracy0.9562\n",
      "Iter204,Testing Accuracy0.9569\n",
      "Iter205,Testing Accuracy0.9591\n",
      "Iter206,Testing Accuracy0.9591\n",
      "Iter207,Testing Accuracy0.959\n",
      "Iter208,Testing Accuracy0.9589\n",
      "Iter209,Testing Accuracy0.9609\n",
      "Iter210,Testing Accuracy0.9575\n",
      "Iter211,Testing Accuracy0.9575\n",
      "Iter212,Testing Accuracy0.9566\n",
      "Iter213,Testing Accuracy0.9579\n",
      "Iter214,Testing Accuracy0.9599\n",
      "Iter215,Testing Accuracy0.9592\n",
      "Iter216,Testing Accuracy0.9581\n",
      "Iter217,Testing Accuracy0.9585\n",
      "Iter218,Testing Accuracy0.9571\n",
      "Iter219,Testing Accuracy0.9594\n",
      "Iter220,Testing Accuracy0.9561\n",
      "Iter221,Testing Accuracy0.9568\n",
      "Iter222,Testing Accuracy0.9577\n",
      "Iter223,Testing Accuracy0.956\n",
      "Iter224,Testing Accuracy0.9574\n",
      "Iter225,Testing Accuracy0.9578\n",
      "Iter226,Testing Accuracy0.9591\n",
      "Iter227,Testing Accuracy0.9569\n",
      "Iter228,Testing Accuracy0.9572\n",
      "Iter229,Testing Accuracy0.9559\n",
      "Iter230,Testing Accuracy0.9587\n",
      "Iter231,Testing Accuracy0.9578\n",
      "Iter232,Testing Accuracy0.9543\n",
      "Iter233,Testing Accuracy0.9572\n",
      "Iter234,Testing Accuracy0.9589\n",
      "Iter235,Testing Accuracy0.9588\n",
      "Iter236,Testing Accuracy0.9577\n",
      "Iter237,Testing Accuracy0.9602\n",
      "Iter238,Testing Accuracy0.9592\n",
      "Iter239,Testing Accuracy0.9594\n",
      "Iter240,Testing Accuracy0.9573\n",
      "Iter241,Testing Accuracy0.9572\n",
      "Iter242,Testing Accuracy0.9591\n",
      "Iter243,Testing Accuracy0.9587\n",
      "Iter244,Testing Accuracy0.9579\n",
      "Iter245,Testing Accuracy0.9604\n",
      "Iter246,Testing Accuracy0.9593\n",
      "Iter247,Testing Accuracy0.9568\n",
      "Iter248,Testing Accuracy0.9587\n",
      "Iter249,Testing Accuracy0.9596\n",
      "Iter250,Testing Accuracy0.9584\n",
      "Iter251,Testing Accuracy0.9589\n",
      "Iter252,Testing Accuracy0.9564\n",
      "Iter253,Testing Accuracy0.9599\n",
      "Iter254,Testing Accuracy0.9587\n",
      "Iter255,Testing Accuracy0.9575\n",
      "Iter256,Testing Accuracy0.957\n",
      "Iter257,Testing Accuracy0.9566\n",
      "Iter258,Testing Accuracy0.9588\n",
      "Iter259,Testing Accuracy0.9576\n",
      "Iter260,Testing Accuracy0.9572\n",
      "Iter261,Testing Accuracy0.9592\n",
      "Iter262,Testing Accuracy0.9588\n",
      "Iter263,Testing Accuracy0.9573\n",
      "Iter264,Testing Accuracy0.9596\n",
      "Iter265,Testing Accuracy0.9556\n",
      "Iter266,Testing Accuracy0.9574\n",
      "Iter267,Testing Accuracy0.9594\n",
      "Iter268,Testing Accuracy0.9585\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter269,Testing Accuracy0.9555\n",
      "Iter270,Testing Accuracy0.9561\n",
      "Iter271,Testing Accuracy0.9604\n",
      "Iter272,Testing Accuracy0.9593\n",
      "Iter273,Testing Accuracy0.9576\n",
      "Iter274,Testing Accuracy0.9552\n",
      "Iter275,Testing Accuracy0.9592\n",
      "Iter276,Testing Accuracy0.9579\n",
      "Iter277,Testing Accuracy0.96\n",
      "Iter278,Testing Accuracy0.9601\n",
      "Iter279,Testing Accuracy0.9596\n",
      "Iter280,Testing Accuracy0.958\n",
      "Iter281,Testing Accuracy0.9578\n",
      "Iter282,Testing Accuracy0.9589\n",
      "Iter283,Testing Accuracy0.9587\n",
      "Iter284,Testing Accuracy0.9578\n",
      "Iter285,Testing Accuracy0.9587\n",
      "Iter286,Testing Accuracy0.9576\n",
      "Iter287,Testing Accuracy0.9561\n",
      "Iter288,Testing Accuracy0.9573\n",
      "Iter289,Testing Accuracy0.9562\n",
      "Iter290,Testing Accuracy0.9585\n",
      "Iter291,Testing Accuracy0.9585\n",
      "Iter292,Testing Accuracy0.958\n",
      "Iter293,Testing Accuracy0.9543\n",
      "Iter294,Testing Accuracy0.9584\n",
      "Iter295,Testing Accuracy0.9533\n",
      "Iter296,Testing Accuracy0.9578\n",
      "Iter297,Testing Accuracy0.9557\n",
      "Iter298,Testing Accuracy0.9584\n",
      "Iter299,Testing Accuracy0.9591\n"
     ]
    }
   ],
   "source": [
    "  # Test trained model\n",
    "correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(prediction, 1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "    for epoch in range(300):\n",
    "        for batch in range(60000):\n",
    "            batch_xs,batch_ys = mnist.train.next_batch(50)\n",
    "            sess.run(train_step,feed_dict={x:batch_xs,y:batch_ys})\n",
    "        acc = sess.run(accuracy,feed_dict={x:mnist.test.images,y:mnist.test.labels})\n",
    "        print(\"Iter\" + str(epoch) + \",Testing Accuracy\" + str(acc))"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "本周作业花费了很多时间，虽然如此也并没有达到想要的效果。因为自身能力不足，无论如何调整，到了96%就很难再往上提高了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 作业总结"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "内容小结：\n",
    "1）通过本周作业对神经网络有了初步的认识\n",
    "2）因为调参过程占据了大部分时间，所以对各类激活函数和学习率在数据集上的表现也有了一定的认识\n",
    "3）虽然本周没有达到作业要求，但是完全是自己独立完成，所以对编程能力也有一定程度的训练\n",
    "4）对提示中的信息还没有进行完全的尝试\n",
    "\n",
    "问题小结：\n",
    "1）在使用relu激活函数的时候，表现比较诡异，准确率始终是0.098，并不知道是什么原因\n",
    "2）当两个隐层都使用elu激活函数时，准确率同样表现为固定的极小值\n",
    "3）理论方面还有需要消化的地方，之后会反复理解\n",
    "\n",
    "因为之前工作较忙，没有系统的时间学习，作业提交也比较晚，希望助教大大谅解。我也会尽快赶上进度，在之后的学习中更加努力。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
